Disclaimer

THESE MATERIALS ARE PROVIDED "AS IS" AND ALLOTROPE EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, INCLUDING, WITHOUT LIMITATION, THE WARRANTIES OF NON-INFRINGEMENT, TITLE, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

Document Conventions

Data Types

When converting data to byte strings, different data types must be handled separately. In the rest of the document, we always specify which data type is used when an object is converted to bytes.

Message Digest Algorithms

ADF uses existing message digest algorithms to convert a stream of bytes to a check sum.

Currently supported are MD2, MD5, SHA-1, SHA-256, SHA-384, and SHA-512. The checksums in ADF (in general and for the DataPackage) are by default intended to ensure the integrity of the ADF file against accidental storage or transfer errors. To do this while providing the best performance, the default algorithm is set to MD5. This algorithm is no longer considered to be cryptographically secure, but for the intended purpose one of the best candidates. However, users can choose a configuration with one of the other algorithms. The checksum service can be configured per ADF file when activating hashing for the file. For DataPackage the algorithm can be set for each DataPackage file created individually.

When we state that data is added to a message digest, we mean that the data is converted to bytes (by the rules above), and then the resulting bytes are sequentially added to the digest algorithm. This must yield the same result as constructing an array of bytes that is filled by the outcomes of the conversions and then running the message digest algorithm on it.

Combining hash values hierarchically

We use a hierarchical approach to combine the hash values of several parts of the ADF file.

In ADF, we use this mechanism at several levels to minimize computation costs.

For the sake of brevity, we use a fictitious digest algorithm, SHA-32, for examples, where we just take the first 4 bytes of the SHA-256 algorithm’s result.

Computation of HDF5 group hash values

The hash value of an HDF5 group is based on its attributes and its child elements (other groups and datasets). The following values are added to the message digest algorithm:

1. If the group is not the root group, the group's name is added as an UTF8 string
2. If the group has any attributes that are not excluded (see below), the UTF8 string attributes is added.
3. The attributes (except for the excluded ones) are sorted by their name, and each is added to the hash by
   1. Adding its name as UTF8 string
   2. Adding its value, depending on its HDF5 type:
      • HDF5 integers (type class H5T.INTEGER whose size is smaller 4 or whose size is 4 and whose sign is not 0) are encoded as integers.
      • HDF5 integers (type class H5T.INTEGER which do not fulfill the conditions of the previous item and whose size is smaller 8 or whose size is 8 and whose sign is not 0) are encoded as longs.
      • HDF5 strings (type class H5T.STRING) are encoded as UTF8 strings.
4. If the group has any child elements, the UTF8 string elements is added.
5. The group's child elements are sorted by their name; for each child which is not excluded (see below) we add:
   1. The name of the child as UTF-8 string
   2. The bytes of the child's hash value. If the child is a group, that value is computed recursively.

Computation of HDF5 dataset hash values

For every non-scalar data set, a check sum data set with the same number of dimensions is generated. It has the same path and name as the original dataset, but is located under the HDF5 group /check-sums. A scalar dataset is one without any dimensions. Such a dataset just contains just a single value.

For every dimension i, a number hashblock_i is chosen that describes the number of elements that are grouped together in that direction to compute a single hash-value.

We denote the size of the data set in dimension i with size_i.

In the hash data set, we store a check sum for each group. Each dimension of the hash data set is of size hashsize_i = size_i / hashblock_i, where we round up (hashsize_i = ceil(size_i / hashblock_i)).

How hashblock_i is chosen is up to the implementation. The chosen values are stored in the HDF5 attribute hash_block_size as a string which contains the comma-separated block sizes.

Each hash value of a block is the result of computing the message digest of the following values:

• The number of dimensions as int
• For each dimension i: the number of elements in the block as long
• All elements of the hashed block according to the type of the data set. The order of the elements is as follows: We start with the first coordinate in every dimension (e.g. (0,0) for two dimensions), then we increase the last dimension first and the first dimension last.
  

The data set used to store the check sums is of data type byte; the last dimension is multiplied by the length of the resulting digest (e.g. 32 for SHA-256) to store all bytes of the hash in the data set.

The overall check sum for the data set is computed by adding the following data to the message digest algorithm:

1. For each dimension, the size of the hash digest is added (as long)
2. We iterate over all coordinates of hashes, starting with the last dimension. For each coordinate, we add the hash contained in the hash data set to the message digest. In case of a scalar dataset there is just one block with one value which is taken directly from the source dataset because no hash dataset has been created.
3. The attributes of the original data set are sorted by their name and for each attribute (except those listed above) we add to the digest:
   1. The name of the attribute (a string)
   2. The value of the attribute (depending on the attribute type, a string, an integer, or a long value)

Computation of the data cube part

The overall hash for the data cubes is computed by adding

• The URI of the data cube (a string)
• The hash value of the data cube (a list of bytes)

Information on the hash is stored in the meta data for the Cube ?Cube with

            ?Cube adf-audit:hasDigest  [
               adf-audit:hasDigestMethod lc-hash:sha256;
               adf-audit:digestValue "..."^xsd:base64Binary
            ]
          

Computation of the data package part

For computing the hash value of the data package, we only consider the data sets of the files, not the directory layout and other meta-data like time of last change. The directory layout and meta-data are stored in the technical model of the ADF file, whose hash computation is described below.

Each file in the data description is backed by an HDF5 dataset. First, we take the RDF representation of all files (resources whose type is adf-dp:File) and sort them by their URI. For every file, we add

• the URI of the file's resource (a string) and
• the hash value of the file's data set (see below).

Computation of the data description part

Several HDF5 datasets are used to store the information. Their hash values are added to the message digest for the computation of the overall data description hash. The following list contains the data sets whose hash values are added, with attribute names that must be included in the hash (see below).

e.g. /data-description/dictionary/bytes.

For each data set in the list, the following data is added to the message digest:

• The path of the data set (as string)
• The size attribute of the group containing the data set (for those data set marked accordingly in the list) (as integer)
• The hash value of the data set

The hash values of the data sets are not stored in the data description itself. They are stored as a hexadecimal string in the HDF5 attribute "CHECKSUM" instead.

Computation of the audit part

Like the data description, the audit store contains RDF graphs. To compute the audit's check sum, we first add the dictionary's dataset's check sums analogously to the data description, only the path is relative to /audit-trail.

Then, all data sets that are included in any audit trail are sorted alphabetically by their URI and for each data set

• The URI of the dataset resource (as string)
• The hash value of the data set (as byte string)

Computing Check Sums of Data Sets

In the current version of ADF, the file format HDF5 is used as an underlying storage solution. Most of the stored information in an ADF file are stored in HDF5 datasets. This section describes how the check sum of a data set is computed.

Because data sets can contain large amounts of data, we apply again the approach of combining multiple hashes to one overall hash.

A data set has n dimensions and a data type. Each entry of the data set belongs to the data type. In ADF, the data types byte, short, int, long, float and double can be used.

When we use coordinates of elements of a data set in the following, we start with 0 (not 1).

        <adf://some_arbitrary_uri> a adf-hash:HashDataSet;
          adf-audit:hasDigest [
              a adf-audit:Digest
              adf-hash:checksumDatasetOf <adf://some_dataset>
              audit:hasDigestMethod [
                  a audit:DigestMethod;
                  adf-hash:hashDimension [
                      a adf-hash:HashDimensionDescription;
                      hdf:order 1 ;
                      adf-hash:hashBlockSize 1000;
                  ];
                  adf-hash:hashDimension [
                      a adf-hash:HashDimensionDescription;
                      hdf:order 2;
                      adf-hash:hashBlockSize 500;
                  ];
                  adf-audit:hasDigestAlgorithm lc-hash:sha-256;
              ]
          ]